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■ Abstract 

We consider the long term behaviour of a Markov chain £,(t) on Z N based on the N 

station supermarket model. Different routing policies for the supermarket model give 

different Markov chains. We show that for a general class of local routing policies, join 

the least weighted queue (JLW), the N one-dimensional components £,i(t) can be par- 

, titioned into disjoint clusters C^. Within each cluster the speed of each component 

. £,j converges to a constant Vk and under certain conditions E, is recurrent in shape on 

(N ■ 

each cluster. To establish these results we have assembled methods from two distinct 
areas of mathematics, semi-martingale techniques used for showing stability of Markov 
chains together with the theory of optimal flows in networks. As corollaries to our main 
result we obtain the stability classification of the supermarket model under any JLW 
policy and can explicitly compute the Ck and V|< for any instance of the model and 
specific JLW policy. 

Keywords: join the least weighted queue, recurrence in shape, network flows, Lyapunov 
functions 
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1 Introduction 

We consider the long term behaviour of a Markov chain £,(t) based on the supermarket 
model of queueing theory. In this model there are N stations, each of which processes jobs 



1 



which queue there. Jobs depart the system after their service is completed. The interesting 
feature is that the stations support a neighbourhood structure of non-empty sets of stations. 
Job streams arrive at these neighbourhoods and upon arrival each job must be routed to a 
station within its neighbourhood. The choice of queue can depend upon the current queue 
lengths. Policies which route jobs based only upon information about the queues in their own 
neighbourhoods are called local. There is considerable interest in the difference in performance 
between systems with multi-station neighbourhoods and those with isolated stations (so no 
routing) motivated by the work of Mitzenmacher and others, see for example [UJ and |12j . 
with some sophisticated asymptotic work by Luczak and co-authors in [5] and other papers. 
The most commonly studied example of a local policy is join the shortest queue (JSQ). We 
consider a generalisation of JSQ where each station j has a weight factor wj > and each 
job joins a least weighted queue (JLW) at a station within its neighbourhood (so JSQ is the 
case where all Wj are equal). 

A simple Markov model, X(t) say, of such a system has independent Poisson arrival 
streams to the neighbourhoods, exponential service times at each station and lives on Z+. 
Our Markov chain E,(t) is based on X(t) but we drop the requirement that the process is non- 
negative so £,(t) lives on Z N (we describe the transition law in detail below). This enables 
us to exhibit behaviour of the process that will only be seen for the queueing model X(t) 
in large deviation situations. While the Markov assumptions are strong we allow general 
neighbourhood structures, arrival rates and service rates and our results are about the long 
term behaviour of finite systems, not large system asymptotics. 

Our main result, Theorem @J says that JLW policies induce dependence between compo- 
nents £,j. Disjoint clusters of stations appear (distinct from but determined by the neigh- 
bourhood structure together with event rates) and at each station j within cluster say 
the drift rate WjE(^(t + 1) - fy(t) | £,(t)) -) V k for some constants V^. It follows that the 
weighted components Wj£,j within a cluster are eventually much closer to each other than 
to those in other clusters. Under some constraints on the neighbourhoods and event rates 
we show in Theorem [5] that the weighted process w£,(t) restricted to a cluster is recurrent 
in shape, an idea which appeared in Andjel et al [T] with some further application in [9]. 
This behaviour is caused by the routing policy and is akin to state space collapse as dis- 
cussed in several queueing network papers studying heavy traffic e.g. Bramson [3] and Kelly 
&; Williams [7] though the time scales and techniques involved are entirely distinct. Our 
results are established with semi-martingale/Lyapunov function methods after the analysis 
of a carefully chosen deterministic flow model on a graph. 
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This preliminary work on flows also leads to two new results for the queueing model 
X(t). Label the clusters and their drift rates so that Vi is the largest such rate. We show 
in Theorem [2] that X(t) is stable when V| < and transient when Vi > 0. This result was 
shown for a system with identical servers in a single neighbourhood by Weber [13] , then for 
a Markov system under JSQ by Foley and MacDonald [B] and then for a system with more 
general arrival streams and service times again under JSQ by Dai et al |4J but we are not 
sure if it is known for JLW. Writing Vi (w) to indicate dependence upon the JLW weights we 
have also shown in Theorem [3] that if Vi (w) < for some positive weights w then V (w') < 
for any set of positive weights w'. In particular if X(t) is stable under any version of JLW it 
is also stable under JSQ. 

To establish these results we compare the behaviour of £, under JLW to that under care- 
fully chosen static policies which also cluster the stations so that the weighted drift rates 
within clusters are constant. In Theorem [T] we show that the cluster structure and drift rates 
Vk can be determined for any neighbourhoods and event rates by solving a particular flow 
problem on a bipartite graph. 

1.1 Model details and notation 

We will mostly consider two classes of simple Markov routing policies described below. All 
jobs are of a single type but the servers have different rates. We assume here that service times 
at each station j are exponentially distributed with rate fx, and that they are independent of 
arrivals and other service times. Each job leaves the system after completion of its service. 
We make no specific assumptions about the queue discipline as we will not discuss waiting 
times of individual jobs but we do assume the servers are non-idling so when there are jobs 
in the queue at station j the departure process is of rate |J.j . 

The stations support a neighbourhood structure of non-empty sets of stations St G V[Co), 
the collection of subsets of Co = {1 , 2, . . . , N}. Jobs arrive at the neighbourhoods as inde- 
pendent Poisson processes with rate \ > at Si for each i = 1 , 2, We allow some \ = 

and denote by A/"(Co) the neighbourhoods Si with At > 0. For simplicity we will usually write 
i G A/ - (Co) when we mean Si G A/ - (Co). To eliminate some trivial situations we suppose the 
bipartite graph G, with nodes A/" (Co) U Co and edges E = {(Si, j) : j G Si}, is connected which 
ensures that model cannot be trivially decomposed into independent components. 

We are interested in the behaviour of the queue length process X(t) on state space N N 
(N denotes the non-negative integers) and a related process, the random walk £,(t) with the 
same jump rates as X at positive states but not reflected at and hence with state space Z N . 
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The exact details of the jump rates depend upon the routing policy so we discuss these now. 

Upon arrival at neighbourhood Si a job is routed to a station j G Si where j is chosen by 
some routing policy. We will mostly consider two classes of simple Markov routing policies 
described next. 

Define A ={pG [0, 1] N : ^ pj = 1} and for each i = 1, . . . , |JV(C )| let 

Ai ={p G A :pj = for j g Si} 
denote the unit simplex on coordinates j G Si. 
Definition 1. A stationary Markov routing policy is a mapping 

tt: Z N x {1, 2, ... ,lA/"(C )|}-> A such that 7t(x, i) G A t . 

Under policy 7t a job arriving at neighbourhood Si when the process state is x is routed to 
station j G Si with probability 7t(x,i)j. We denote the space of routing policies by IT. 

If 7i does not depend upon x we say it is static and write 7r(i) for the routing distribution 
of arrivals at Si. We denote the space of static routing policies by rT stat . n 

Definition 2 (Local routing policies). Fix a set of positive weights {wj : j G Co,Wj > 0}. 
For x G Z N let wx i = mini eSi wvXv and let Bi{x) = {) G Si : WjXj = wx i j denote the set of 
stations in Si with minimal weighted state value. The join the least weighted queue (JLW) 
routing policy is defined by 

. / 1/|£i(x)| ifjGBi(x), 

[ otherwise. □ 

Remark 1. The JLW policies are stationary Markov but not static. They are of practical 
interest as they are local and relatively simple to implement (only the queue lengths in an 
arrival's neighbourhood are needed to make its routing decision). There are several varieties 
of JLW (making different choices when |£h(x)| > 2) and close variants like routing to stations 
where w, (x, + 1 ) is minimal but the system behaviour at the level considered here is much 
the same for all variants. 

Two particular cases have been studied for a variety of models. With Wj = 1 for each j 
the policy is join the shortest queue (JSQ). With weights Wj = the policy is join the 
smallest workload (JSW). Another plausible choice of w is to give most stations weight 1 but 
protect some stations by making their Wj larger. □ 

The jumps and jump rates of the queue length process X and related random walk £, 
under stationary Markov routing policy 7t are as follows. The possible jumps change the 
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current state x by ±ej, the unit vector in M N with value 1 in component j. Both processes 
make up-jumps (a job arrives and is routed to a station) with the same rates from all states 
x G Z N i.e. we have x i— > x + ej at rate 2^vjes t \ n i x ) Downward jumps (job completions) 
x i— > x — ej occur at rate [ij at all states x G Z N for the random walk and at all states x G N N 
with Xj > 1 for the queue length process X. 

Under routing policy n the drift rate V of £,j(t) at every state x and of Xj(t) at x with 
Xj > satisfies 

V(j;x,7t)= Y_ Ai7t(x,i)j-^ (1.1) 

ieMCo) 

which simplifies to V(j;7t) — Y.ieAf{c ) ^i n (i-)) ~ Mi f° r static policies. 

With our interest in JLW policies it is convenient to work with weighted versions of X 
and £,. Let w = (wi , . . . , wn ) be a set of positive station dependent weights. We will write 
wx for the vector (wiXi, . . . ,w N x N ) for each x G Z N and with this same convention write 
wX and w£, for the weighted processes. 

Remark 2. For any set of positive weights w and under any policy n the processes X and wX 
can be coupled so both reach state at the same times and hence both processes are recurrent 
or transient together under any fixed policies. We discuss stability of X under different local 
policies below. □ 

Remark 3. Neighbourhood-station interaction. We have restricted our attention to station 
dependent service rates ^ here as models where jobs arriving at Si have service rates 
when routed to station j show behaviour that is far from optimal under local routing rules 
like JLW. 

Consider the following simple example. There are 3 stations {0, 1 , 2} and 3 neighbourhoods 
Si = {i, [i + 1]}, for i = 0, 1 , 2 where [i + 1] = i + 1 (mod 3) and the service rates for each 
i are jj,^ = 1 for j = i but (Xy = 1 /2 for j = [i + 1]. The arrival rates are Ai = 0.7 for each i 
and the policy that routes all Si arrivals to station i for each i is clearly stable and in fact 
minimizes the drift rate at each station. From the symmetry of the situation JSQ (with ties 
split 50/50) sends half of all arrivals to the station where they receive the slow service rate 
so the long run average service time is (1 + 2)/2 = 1 .5 and as 0.7 x 1.5 > 1 the system will 
be unstable under JSQ. □ 

1.2 Results 

While our main interest is in local routing policies we start with some results for static routing 
policies for the queue length process X. To link them to the JLW policy we fix upon a set of 
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positive station dependent weights w. We are interested in static policies that stabilize the 
system when this is possible i.e. policies that keep the weighted drift rates (11. ip small in the 
following sense: 

• the maximal drift rate is as small as possible, 

• the number of queues growing at maximal speed is minimal, 

• the second largest drift rate is minimal on a minimal set of queues and so on. 

We will refer to any non-empty collection of stations C C Co as a cluster to separate 
it from association with any particular arrival streams. For any cluster C and any class of 
static policies TT' C TT stat define 

V(C;w,TT') = minmax-WjV(j;7t) (1.2) 

7ten' jec 

i.e. the minimum (over policies in TT') of the maximum drift rate of weighted jobs over stations 
in C. 

Theorem 1. Let V| = V(Co;*w, T\ stat ). We can decompose the set of stations into a hierarchy 
of disjoint clusters C], . . . , Ck for some 1 < K < N with the following properties. 

(i) Ci is the unique cluster C such that V(C; w, T\ sta t) = V] and |C| is minimal. 

(ii) If Ci 7^ Co then for stages k = 2, . . . let 

Vic = V(C \U^- 1 C n ;w,n k _ 1 ) 

where TT^ i is the set of static policies that achieve V n on cluster C n for n = 1 ; . . . , 
k — 1. Ck C Co \ U, C n is the unique cluster that satisfies V(C;w, TTk-i) = Vk with 
minimal value of |C|. At each k we have Vk < Vk-i . 

For some K < N ; ufQ = Co and the hierarchical minimax decomposition is complete. 
(Hi) TTk is non-empty. For any n e TTk and each j G Ck, k = 1 , . . . , K 

w i ( Y- ~ Mi) = v k • 

ieAf(Co) 

It turns out that the clusters Ci and drift values Vj. tell us a great deal about the behaviour 
of the queue length process X(t) and the random walk model £,(t) under the JLW policy 
with the weights w used in (11.21) . Under any static policy the queues at each station are 
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independent and for n £ TTk the drift rate at each j £ is V^. Under a JLW policy the 
queues will not be independent in general. 

The next result concerns stability of X(t) which for JSQ applied to this Markov model 
first appears in Foley and McDonald [6J and was extended by Dai et al [4j for a model with 
non-Markov arrival processes and non-exponential service. 

Theorem 2. Choose a set of positive weights w. If Vi > then the queue length process 
X(t) is transient under any policy. IfV] < then the queue length process X(t) is positive 
recurrent under the JLW policy with weights w and under any n £ TTk- 

While this stability condition is known in the JSQ case (all = 1 ) our approach via the 
hierarchical minimax decomposition with static policies allows us to compare the stability of 
X under different local routing policies. 

Theorem 3. Let Vi (w) denote the maximal drift rate obtained from the hierarchical minimax 
decomposition with weights w. IfVi[w) < for some positive w then V](w') < for all 
positive weights w'. Thus if X is stable under any JLW policy it is stable under all such 
policies. 

Next we consider the dynamics of the random walk process £, under local policies with 
any fixed choice of weights w. The JLW policy makes the queue workloads within clusters 
dependent and our main result is that can be interpreted as the rate of change to the 
weighted queue length at each queue in cluster Ck under JLW routing. This is a new obser- 
vation for this model. In stable cases (Vi < 0) this behaviour of the queueing model X(t) 
will only be seen in large deviation situations due to the reflection of the process at 0. This 
is why we have introduced the random walk £,. 

Theorem 4. The random walk £,(t) under the JLW policy eventually displays the hierarchical 
minimax structure. Specifically the weighted random walk w£,(t) eventually has drift rate Vk 
on each cluster Ck, that is for small enough £ > and any finite initial configuration £,(0) = Xo 
there exists a random time t(e) such that for each cluster Ck and for any t > t(e) 



Vv 



< t £ for each ) £ Cj< 



t 

Remark 4. In cases where Vk > for some clusters Ck this result can be extended to apply 
to the queueing model X(t) on such clusters. □ 

Under some slightly stronger conditions on the internal structure of the clusters we find 
that the JLW policy causes w£, to exhibit some remarkable behaviour which we call recurrence 
in shape. 
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Definition 3. For any Markov process Y(t) on Z N let Y c (t) = (Yj(t) : j G C) denote the 
process (perhaps not Markov) on components in C for any C C {1 , 2, . . . , N}. For any jo G C 
we say that Yc(t) is recurrent in shape (or Y(t) is recurrent in shape on C) when the process 
(Yj(t) — Yj (t)) : j G C) is recurrent. □ 

This notion was applied to a single cluster storage model in [9]. To apply it here we need 
a slightly more refined description of the internal structure of the clusters. For n G TT sta t let 
G(7t) denote the bipartite graph with nodes M{Cq) U Co and edges E = {(Si, j) : 7t(i)j > 0}. 

Definition 4. Cluster C C Ck is bonded if there exists a policy 7r G TT k such that for every 
pair j, m G C there is a path in G(7t) from j to m. □ 

Theorem 5. Under the JLW policy with weights ~w the weighted random walk w£,(t) is 
recurrent in shape on each bonded sub-cluster of C k; k = 1 , . . . ; K. 

The restriction to bonded clusters is necessary as in general a cluster can split into two 
or more independent parts with the same drift rates. 

The clustering behaviour described here is probably also exhibited by the supermarket 
model with more general arrival streams and non-exponential service times and we are looking 
for Lyapunov functions that will allow us to extend our arguments at the necessary points. 
It is possible that similar behaviour will persist in similar systems which have Jackson-style 
feedback though there are many complications here, see for example Dai et al [1]. In fact it 
was in trying to understand [I] that we discovered the results described here. 

2 Proofs 

2.1 Preliminaries for Theorem [I] 

We now describe a flow based decomposition of the system based on some ideas from the max- 
flow, min-cut theorem of Ford and Fulkerson for models of flow in networks. Then we show 
that the hierarchical minimax decomposition coincides with the flow based decomposition. 
In this section we only consider static routing policies and we work with a fixed set of positive 
weights w as before. 

Flow based decomposition Recall that any non-empty set of stations is called a cluster. 
For each cluster C C Co let P(C) be the collection of subsets of C and M{C) = {i G Af[Co) : 
Si C C} the collection of neighbourhoods with Ai > supported by C. The system restricted 
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to C consists of the stations in C with the arrival streams to neighbourhoods in Af{C). Such 
a restriction can be achieved by applying a routing policy from 

TT(C) ={ne n stat : 7t(i), = if i g Af(C), ) e C}, 

the set of static policies which are consistent with this decomposition. 

We also introduce the idea of the system reduced onto a cluster D by removing the 
stations in Co \ D together with the arrival streams to neighbourhoods in A/"(Co \ D). For 
each non-empty S G V{D) define 0s(D) = {Si G Af(Co) : Si fl D = S} i.e. the collection of 
neighbourhoods (if any) that coincide with S on D, and merge all arrival streams in Os(D) 
to get one with rate 

A S (D)= Y_ A i- (2- 1 ) 

veers (D) 

Now for C C D let Ad(C) = {S G V{C) : As(D) > 0}. Combining these two notions we 
see there are static routing policies which act to decompose the original system into one 
restricted to a cluster Ci and an independent system reduced onto D = Co \ Ci which can 
be further decomposed as desired. These steps can be repeated to sequentially decompose 
the system. 

We must also consider the drift rates at stations under any such decomposition. For the 
system reduced onto D define, for any cluster CCD, the average D -reduced restricted drift 
on C by 

Wd(C) 4 ^ MD)Tt(i), - n,) ] . (2.2) 

1 1 \ jec ieA/- D (C) / 

We will not change the weights w = (wj) during the decomposition so we do not indicate 
Wd's dependence upon w. 

Remark 5. The conditions W D (C) < for every cluster C C D are closely related to the 
sufficient conditions for a matching on a bipartite graph. In the special case where each 
Wj = 1 we can reverse the order of summation above and sum out the dependence on n to 
get 

W D (C) = — ( Y_ MD)-^) . 

ieA^ D (C) jec 

Hence W D (C) < only when the total service rate of servers in C is at least as large as the 
total rate of arrivals to neighbourhoods entirely supported by C. □ 

Jobs arriving at neighbourhoods in A/"(C) cannot be routed to stations outside C but it 
may be possible to route jobs from other neighbourhoods to stations in C so, by comparison 
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with fll.ip . for any policy n at any x where n is consistent with reduction onto D and for any 
C C D we have 

^ Wj V(j;x,7t) = ^ Wj ( Y_ MDMx,!),-^) >|C|W D (C) (2.3) 

jGC jGC iSA/o (D) 

i.e. the D-reduced restricted drift is a lower bound for the average drift over C under any 
policy 7r on the D-reduced system and this lower bound is reached by some policies. 

There is one further small result which it is convenient to separate out from the proof of 
Theorem [TJ 

Lemma 1. Suppose A, B are clusters contained within D such that 

W D (A) = W D (B) = maxW D (C) =v. 

CCD 

LetAf, = A/"d(A UB)\ (Ad (A) U A/"d(B)). TknW D (AUB) =v and ^ MD) = 0. 

Proof Let H = A n B and A/" 2 = AT D (B) \Ad(H) and note that jV D (B \ H) C AT 2 . Let 
LT(H) V denote the set of policies that achieve Y.jen w i (2Zaa d (h) MD)7i(i)j — (x,-) < |H|v. By 
maximality of v we have Wo(H) < v and hence TT(H) V is non-empty and in fact it contains 
policies that achieve the D-reduced restricted drifts for the clusters A and B. 
For any policy 7t we have 

|B|W D (B)<^ Wj Y_ MDM^ + ^Wj^MDMi);-^™^. 

jGH A/d(H) jeB ieA/" 2 jeB 

For any policy n G TT(H) V it follows that 

|B \ H|v < ^ Ai(D)7i(i)j — ^Wj|j.j 

j£B ieA/" 2 B\H 

and from this we have, for any n G TT(H) V , 

jeAUB ieA/" D (AUB) 

jeA ieA/" D (A) jeB ieA^ 2 

jeB\H jeAuB ieA/i 

> v(|A| + |B \ H|) + ^ Wj ^ \t(D)7t(i), . 

jeAuB ieM 

By maximality of v and |AUB| = |A| + |B\H| we see that Y.M Ai(D) = and W D (AUB) = v 
as required. ■ 
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2.2 Proof of Theorem U 

The routing schemes we consider here minimize the maximum drift by directing some arrivals 
from heavily loaded parts of the network to less loaded parts. Part (i) of this Theorem deals 
with the most heavily loaded part first. 

(i) Using the flow based scheme we define at stage 1, D = Co and 

W| = maxWb(C) , C,=U{CCD: W D (C) = W} 

and we now show that Wb(C|) = Wi . If there is only one C with Wd(C) = W] we 
are done so suppose there are more and take any two distinct clusters C, C such that 
W D (C) = Wb(C') = Wi. By Lemma [H W D (C U C) = Wi from which it soon follows that 
Wb(Ci) = W] and hence Ci is the unique maximal cluster with restricted drift value Wy. 

Next we must relate Wi (an average drift on a cluster) to V| (a maximal drift within a 
cluster). We start by showing that there exist policies n' £ TT(Ci) such that WjV(j;7t') = W| 
for each j £ Ci. If not then let TT(Ci )* C TT(Ci) denote the set of policies n that achieve the 
restricted drift Wi i.e. 

^w j V(j; 7 t) = |Ci|Wi . 

Ci 

Next pick ft £ T7( Ci )* that achieves "0 = min n ^ y, maxj g ^ WjV(j;7t) at some stations in Ci. 
By assumption *v> > Wi. Let C = {j £ Ci : WjV(j;ft) = "0} and consider the restricted drift 
Wd(C) on C. As "0 is minimal any calls that can be routed out of £ by ft will be and so 
"0 = Wd(C) but this implies Wd(C) > W| so by maximality of Wi we must have "0 = Wi. 

Hence there exist policies n' £ TT(Ci)* such that WjV(j;7t / ) < Wi for j £ Ci. As 
^jgc, w jVly, n ') = |Ci|Wi for such policies we must have WjV(j;7t') = Wi for each j £ Ci as 
required. By maximality of Ci we cannot have WjV(j;7t') = Wi for all n' for any j ^ Ci. 

Now pick ft £ n(Ci) such that WjV(j;^) = W, for all j £ C, and WjV(j;^) < W for 
j ^ Ci . Then 

Vi = min maxWjV(j;7t) < maxWjV(i;ft) = Wi 

7ten st at jeD jeD 

but from 02. 3p we have, for any n £ n stat , 

maxWjV(j;7r) > -=L- WjV(j;7t) > Wi . 
,£D ' Cl 'jec, 

Hence V| = Wi and no policy with maximal drift rate Vi can achieve drift rate less than Vi 
at any station j £ Ci. As WjV(j;ft) = V for j £ Ci, WjV{y,ft) < V for j ^ Ci it follows that 
Ci = Ci . This completes part (i) of the theorem. 
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(ii) & (iii) For systems where Ci ^ Co we can continue for stages k > 2. At stage k. reduce 
the system onto D k = Co \ U^ 1 Ci and define 

W k = maxW D JC) , C k = U{C C D k : W D JC) = W k }. 

CCD k 

At each stage V k = W k and C k = C k follow as in stage 1. That V k < V k _i follows from the 
restricted drifts for if V k > V k _i then W Dk _, (C k _i U C k ) > W Dk _, (C k _0 and |C k _i U C k | > 
|C k _i| in contradiction to |C k _i| being maximal. 

After a finite number of stages K (K < N, the number of stations) we will have Ck = Dk 
which completes the decomposition. ■ 

Remark 6. (conservation of mass) Consider static policies n that are consistent with reduc- 
tion of the system onto D = U^Q and route all flow possible out of C k . If under such a n 
we have, for each j G C k , Wj (^jvbfc ) ^i(D)7t(i)j — M-j) = V where V is constant then V = V k . 
To see this sum the equations Wj (^ieMatc ] ^i(D)7r(i)j — = V over j G C k to get 

jec k jec k iGAA D (c k ) 

A/d (C k ) C k C k A^ D (C k ) C k 

This equation does not depend upon n and is satisfied by V k . We establish the same result for 
more general policies in the proof of Theorem H] but there is no equivalent result for models 
with service rates that depend upon the routing decision Si to j. □ 

2.3 Proof of Theorem H 

For any static routing policy 7t, the arrivals to stations 1 , . . . , N are independent Poisson 
processes, arrivals to station j having rate Y-ieN{c ) ^ n ^))- The system is simply a collection 
of N independent M/M/l queues so it is ergodic under n G TTk if and only if all V k < and 
this is implied by V| < 0. 

Under local policies the queues become dependent and we use Lyapunov or test function 
results to establish transience or recurrence properties. We briefly state a couple of well 
known results that we use a few times in what follows. 

Theorem 6. Suppose {X n } is an irreducible Markov chain on a countable state space S and 
f : S R+ . Let Af n = f (X n+1 ) - f (X n ) . 

(%) If there are constants c > 0, d > and £ > such that |Af n | < d a.s. and 
E(Af n | X n = x) > £ for all x G {x : f (x) > c} then {X n } is transient. 
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(ii) If there is a constant e > and a finite set Ac5 such that E(f(X n+ i) j X n = x) < oo 
for x G A and E(Af n | X n = x) < — £ for x G S \ A then {X n } positive recurrent. 

Proof: part (i) is a special case of Theorem 2.2.7 in [3] - note the need for bounded jumps. 
Part (ii) is Foster's criterion which can be found in many places, for instance Theorem 2.2.4 
of [5] or Proposition 1.5.3 of [2J. ■ 

Now we return to the proof of Theorem [2j If Vi > we show transience under any policy 
7t by using the Lyapunov function L(x) = Y WjXj with X(t)'s jump chain X n . The rate 
of events for X(t) at state x is given by a(x) = Y-M(c ) ^ + 2Zc tVi*^ ) which is bounded. 
Using the notation for the state with every Xj = 0, 1 for the state where every xj = 1 we 
have 

a(0) = ^ h< cx(x) < Y_ Xi + Y- ^ = a ^ • ( 2 ' 4 ) 

AT(Co) AT(C ) C 

Let AL n = L(X n+ i) — L(X n ). We have |AL n j < max, w, at all states of the system. For any 
policy n, inequality (I2.3P applied to cluster Ci with D = Co implies 

ot{x)E n [AL n | X n = x) = Y_ WjV(j;x,7r) > |d|Vi >0 

jec, 

at every state x G N N . By Theorem [^i) the jump chain X n is transient under ix and hence 
so is the queue length process X(t). 

It remains to show that X(t) is stable under the JLW policy when Vi < 0. As the event 
rates lie in the interval [oc(0), <x(l)] we can work with the jump chain X n instead. It is 
convenient to work with a quadratic Lyapunov function here. 

Let q(x) = jX J Qx where Q is a real, symmetric N x N matrix and let e, denote the unit 
vector with = 1 . For 6 G {— 1 , 1 } we have 

q(x + 6ej) - q(x) = 6e[Qx+ ^Qjj 
and we need to compute E^fAqn | X n = x) where Aq n = q(X n+1 ) — q(X n ). With indicator 

functions Dj — 1 {departure from station ]}i A^ 1 {arrival at Si}? Rvj arrival routed to j} We have 



Aq n = ^DjJqtXa-ejJ-qtXn)] + ^ A^ [q(X n + e,) - q(X n )] } 

jGC ieJV(C ) 

Y_ A l Rg-D J )^ T QX a +i2lQfl( L AiRg + D,). (2.5) 



iec ieAT(Co) jec ieA/"(c 

Also, for any policy 7t, we have 

a{x)E n {D i | X n = x) = ^jl{ Xj >o}, aME^A^ | X n = x) = Ai7t(x,i)j . 
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For the specific function S(x) = | Y.)ec w j x f we nave Qjj = w j an d e^Qx = WjXj. Further, 
for 7t G FTk, j G Ck we have Wj f^Zjv^Co) Ai7t(i)j ~ M-j 1 = "W and so for n G TTk, 



cxME^ASn | X n = x) = Y_ Vk Y_ x i + 

k jec k 

(any coefficient is OK for terms Xj = 0) where AS n = S(X n+1 ) — S(X n ) and 



(2.6) 



jec 



As each Vk < the process S(X n ) is a good supermartingale at all but a finite subset of N N 
under any 7t G TTk. 

We complete the proof by comparing the behaviour of S(X n ) under JLW with its behaviour 
under n G Fix- Let El denote expectation under the JLW policy Only the variables Rjj are 
controlled by the routing policy so by comparison with (12. 5p 



<x(x) (E L (AS n I X n = x) - E^ASn | X n = x) 

= oc(x) Y_ w j (*j + 1 /2) Y. f E L( A i R ij I X n = x) - E^A^j | X n = x) 



jec ieA^(Co) 

a(x) Y T- w i x i [ E L(Ai R ij I X n = x) - E^A^ | X n = x) 

ieM{C ) jes t 



+ -!rY w i T. [E L (A i R ij |X n =x)-E 7r (A i R ij |X n = x) 

jec ieM(Co) 

For each neighbourhood recall that wx , = minj eSi WjXj and let i^i = maxj eS . Wj. For the 
second part of this sum we have the simple bound 



oc(x) Y w > T- [ElCA^R^ I X n = x) - E 7t (A i R ij | X n = x 



jec ieAT(c 



II 



IfiiWI 



1{ Wl . Xl -^wxj - y~ Wj ^~ A i 7t(i) j 

Co A^(Co) 



< 



jGS t 



which does not depend on x. We now show the first part of the sum is negative. The JLW 
policy routes arrivals to stations in i3t(x) C Si where we have w,Xj = wx ,. Hence 

cx(x) Y H w i x i E L(AiRij | X n = x) - E^A^ \X n =x) 
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= a(x) Y_ (pOk + ( w i x ) ~ y^S) Y- [ E L( A i R ij I X n = x) - E^AiRtj | X u = x) 

A/"(c ) jeSi 

= Y_ wx^Ai - Ai) + ^ ^ fwjXj -wxj(0- Aj7t(i)j) < 

with strict inequality except when the WjXj are all equal or 7t(i)j = outside £h(x) for every 
St. Combining this with (I2.6P where n € TTk we have 

a(x)E L (AS n | X n = x) = a(x)E 7r (AS n | X n = x) 

+ cx(x) (e l (AS u I X n = x) - E^ASn | X n = x)) 

< ^ V k ^ xj + ^ ^ A^t + Y_ WjMj1{x,>0} 

k jeC k AT(Co) c 

and as Vk < for each k and Xj > for each j the process S(X n ) is a good supermartingale 
under JLW at all but a finite subset of N N . Now Theorem Efii) implies that the jump chain 
X n is positive recurrent and the ergodicity of X(t) under JLW now follows from boundedness 
of the event rates a(x) as before. ■ 

2.4 Proof of Theorem [3] 

In this result we are comparing behaviour of the process under policies defined with different 
sets of weights so we explicitly mention dependence upon w in this section. 

Suppose that for some set of weights w we have Vi (w) < 0. Denote by ft(w) a static 
policy that achieves drift rates Vk(w) on clusters Cic(w). Using the observation in Remark 
[2] we see that the processes X(t) and w'X(t) (for any positive weights w') are also positive 
recurrent under static policy ft(w). Now consider a policy ft(w') that achieves the hierarchical 
minimax rates for weights w'. By definition 

Vi (w') = maxw- ( ) Aift(w'; i)j — \i; ) < maxw- ( ) Ai^(w; i)j — |Xj ) < 
jec V * — / jec V * — / 

and so X(t) is positive recurrent under the static policy ft(w') and hence, by Theorem [21 also 
under JLW with weights w'. ■ 

2.5 Preliminaries for Theorems [4] &; [5] 

The results of these Theorems are for the random walk £,(t) which is obtained from the queue 
length process X(t) by not reflecting it at 0. The first result in this section is a calculation 
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that helps us deduce that JLW ensures that all stations in a single cluster have the same 
drift rate of weighted queue length. 

The overall event rate at all states is <x = Y.Af(c ) ^ + 2Zc Recall that for the process 
reduced onto D = U^Q by any static policy tc G TTk we have to merge neighbourhoods in 
collections o\(D) and sum the relevant flow rates to get total flows A|(D) so, under n G Ilk, 
the event rate at j G Ck is (Xj = (Xj + X.ieA/b(D) ^t(D)7t(t)j. 

We say that x is properly clustered if for each cluster Ck and each j G Ck, WjXj < WiXi 
for each I G C n and WjXj > WiX; for each I G u£ +1 C n . At a properly clustered x the 
event rate at station j 6 Ck under JLW is (Xj{x) = + Y-ieN (d) ^(DlljeB^/l^iMI with 
D = u£Q as for 7r G TTk and £h(x) = {j G Si : WjXj = min leSi is the set of JLW routing 
choices for an Si arrival when the system state is x. 

As before it is convenient to work with the jump chain, this time £, n for the random walk 
£,(t). For each cluster Ck we will study the process Fk(£n) where Fk is the quadratic function 

T f , 1 X- V" (WiXi-WrXr) 2 1 T r\ /q V N 

FkW = -> > = T x'Qx 2.7 

4 L — z — Wiw r 2 

iec k rec k 

where Q lr = — 1 for I ^ r and Q Tr = w r Y-u* 1 / w i- We write AF k (n) = F k (f,n+i) — F k (£, n ) 
and y k = L re c k Vw T • 

Lemma 2. Consider the embedded chain w£, n and the process F] C (£, n )- T/ien 
(z^ /or any 7r G TTk, 

aE„^AF k (n) | £, n = xj = - aj(y k Wj - 1 ) ; 

jec k 

(ii) for any properly clustered state x 

aE L (AF k (n) | £, n = x j < - Y_ M x HYkWj — 1) ■ 

jec k 

Proof of Lemma [2] (%) The effect of any policy n G FTk is to produce independent random 
walks at stations j G Ck with the same drift rate ^V]{Y.ieAf n {c v ) )7r(i)j — (j.,) = V k so the 
calculation for this part is very similar to that for a zero drift random walk with independent 
components. 

To re-use (I2.5P we first calculate e^Qx = ykWjX, — 2Zrec k Xr - Then observe that Qg = 
for the vector g = (l/w r ) Te c k so Fk is constant in this direction and we can translate any 
given x in direction g so that 2Zc k x j = ^- After such a translation we have e^Qx = YkWjXj 
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when ^ Ck x T = and also Qjj = y k Wj — 1 . Taking expectation under n of ( 12. 5ft we have 

otE^AFju) | £, n = xj = Y k ^WjXj( Y MDMi)j ~ Hj) + ^ Y K j(YkWj - 1) 

jec k ieAA D (c k ) jec k 

= TkV k Y Xj + - ^ <Xj (y k Wj - 1 ) 

jec k jec k 

and as X.jec k Xj = we have established part (i). 

(ii) We now consider E L (AF k (n) | f, n = x). As x is properly clustered JLW will only route 
arrivals at neighbourhoods in A/rj(C k ) in the D-reduced system to stations in C k . As in the 
proof of Theorem [21 but now with e^Qx = y k WjXj when Y.c k Xr = ^? 

a E L (AF k (n) | £, n = x) - E 7t (AF k (n) | £, n = x) 

= YhI>jXj[ Y. A i (D)(^M-7t(i) j )] +I^( £ x j (x)-a j )(y k w j -1 

jec k ieA/D(c k ):jeSi lWI jec k 

ieNolCk) jeSi jec k 

since Y.]eSi ''jeSiM = |£>i(x)|. Combining this with the result of part (i) 

1 



cxE L (AF k (n) | £, n = x) = y k Y A ^ D ) ~ Y 7t ^)i w i x i) + J a j( x ^ kW i ~ 1 ) 

iGAT D (c k ) jeSi jec k 

< ^ Y "jMCw^ - 1) 
jec k 

as wx t < ^j gS . 7t(i)jWjXj for each St at any x and for any n. m 
The next lemma is used in the proof of Theorem H] when we show that clusters separate 
apart under JLW. 

Lemma 3. Consider the D-reduced random walk £, restricted to cluster C under JLW routing 
and (i) let C + (n) denote the walk when additional arrivals (not from Aft>{C)) must be routed 
to stations in C; (ii) let C~(n) denote the walk when some arrivals to A/d(C) are routed 
elsewhere. Suppose C + (0) = £~(0) = £,(0). Then C~(ri) < £,(n) < £ + (n) for alln. 

Proof of Lemma [3] By x < z we mean Xj < Zj for each j and as all the Wj > we 
have wx < wz equivalent to x < z for all x, z 6 Z N . We make no assumptions about any 
additional arrivals or arrivals routed elsewhere except measurability of the overall process. 

Suppose C + (0) = £(0) and construct C + (n) from the same down jumps and arrival events 
as £, together with the additional arrivals. Let J(i, x) denote the station chosen by JLW for 
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an S| arrival in state x. We must couple the routing processes also. In particular if station 
j = J(i, x) is chosen at stage n for £, and C + (n)j = Xj then the same station must be chosen 
for C + - This will work as long as C + (n) > £,(ti) for every n > which we now show by 
induction. 

Departures affect each process identically so cannot change order. At stage n write 
x = £,(n) < z = C + (u). At any additional Si arrival set £,(n + 1 ) = x and £ + (u + 1 ) = z + 
where j — J(i, z). At a standard Si arrival, if J(i, x) = J(i, z) = j then £,(n+ 1) = x + ej < 
z + ej = C + (n + 1 ). If J(i, x) = ) ^ 1 = J(i, z) then WjXj < W;X; < W;Zi and WjXj < WjZj 
(if WjZj = WjXj the coupling above forces J(i, z) = j) and again x + ej < z + Z\. Hence by 
induction £,(n) < C, + [n) for all n. 

The argument showing that C~(u) < £,(n) is essentially the same but the routing coupling 
required is that if station j = J(i, z) is chosen at stage n for £~ and £,(n)j = Zj then the same 
station must be chosen for £,. ■ 

Remark 7. This result does not extend to the queue process X because the departure process 
for X is dependent upon the arrival process due to the emptying of queues. □ 



We establish the result for a single cluster using Lemma [2] and an inequality which we state 
next. Then we use Lemma [3] to extend it to successively larger numbers of clusters. 

The following generalization of Kolmogorov's maximal inequality is Lemma 3.1 in [10J. 

Lemma 4. Let (Y t ) t gz+ be a stochastic process on [0, oo) adapted to a filtration {J r t )t€Z+ (f or 
example a function of a Markov chain). Suppose that Y = yo and for some b e (0, oo) and 



Now we continue with the proof of Theorem HI Suppose the hierarchical minimax static 
policy results in a single cluster. We show first that for the embedded chain |Wj£, n j — Wj,£, n ],| < 
n ] ~ E eventually along any sample path and we deduce the result from this. 

We again use the Lyapunov function Fi introduced in (12. 7p . We have maxj ; v e c, IwjXj — 
WiXi| 2 < 2w 2 Fi (x) at any state x, where w = maxc, Wj. Additionally max n /2< T < n Fi (£, T )/r 2 < 



2.6 Proof of Theorem [4] 



all t e z 



E(Y t+1 -Y t | Ft) < b a.s. 



Then for any x > and any positive t G Z 
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4max n /2< r < n Fi (£, r )/n 2 along any sample path as Fi (x) > 0. Let 



< m 



max max 



> 2 



1 - , <r<2 m j,leCi 

Thus 

PL(A m )<P L ( max —£zL>-—)=-pJ max F,^)^-^- 

\2 m - 1 <r<2 m 2 Zm 4 / \2 m - 1 <r<2 m OW z 

We know from Lemma [2] that under JLW El(AFi(ti) | £,(n) = x) < y where y > is a 
constant. By applying Lemma H] starting from £,o = Xo we now have 

, 2 2rrL ( 1 - i; ) 
PL(Am) < Pl max F] (£, T ) > 



< 



" V0<r<2 m 8"& 2 

8rt> 2 (F 1 (x )+y2 m ) 

22m(1-e) 



and for e < 1 /2 Borel-Cantelli implies only finitely many of the A m occur. For any n > 2 
we have 



J Wj£,rj -Wi£, Tl ) _ £ 1 n 

< max max — - — > r > C A 

I r>n j,iec k r J I 1 



m>log 2 n 



,1-2e 



which means that for e < 1 /2 there exists n (e) (random) such that |wj£, n j — Wv£, n v| < n 
for all n > n . 

Returning to the continuous time process £,(t) its event rate is bounded, see (12.41) . so this 
result for the jump chain implies that all components Wj£,j(t), j 6 Ci, eventually have the 
same drift rate. 

Now we show that if all weighted queues in a cluster, say, have the same drift rate it 
must be the rate obtained under the hierarchical minimax policy. Choose T large enough that 
maxj^gCk |wi£,i(0) — Wj£,j(0)| is small compared to T and consider any policy n that achieves 

maxK^m-w^m^T 1 - 6 

for some small £ > 0. For each j e this implies there exist constants V, |3, with ||3j| < 1 
for each j such that Wj (£,, (T) — (0) ) = VT + PjT 1 E . Dividing through by w,T and summing 
over j G Ck we have 

fIft(T)-5j(0))=v£l + r£i. 

T z — v ' L — Wi L — W; 

jec k jec k ' jec k ' 

For large T the left hand side is approximately X.ieA/" D (c k ) ^i(^) ~~ 2^jec k Mj wm ^ e the right 
hand side is approximately V^j e c k ^ / w j anc ^ hence V — » as T — > oo by Remark [6j 
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It remains to show that JLW eventually separates the clusters from any starting con- 
figuration. We start by considering systems where hierarchical minimax routing produces 
two clusters Ci and C2. The optimal static policies route all arrivals at neighbourhoods 
Si £ A/"(Ci) to stations in C2 while arrivals at Si G J\f[C-\) must be routed (by any policy) to 
stations in Ci . The only cluster level routing error JLW can make is to route some arrivals 
at St £ JV(Ci) into d. 

Now we employ Lemma [3j This tells us that all weighted queues in cluster Ci eventually 
have speed at least Vj. Also, while the cluster structure on Cz may be totally changed by 
the lost arrivals, no weighted queue there has speed greater than V2 < V] and so there exists 
a finite (random) time to such that for all t > to, Wj£,j(t) > wi£,i(t) for every pair j G Ci, 
I G Cz- For t > to the process wf,(t) occupies properly clustered states and so JLW no 
longer makes cluster level routing errors. Now the results for single clusters imply that for 
each j G d, Wjf,j(t) has asymptotic drift rate for k = 1, 2. 

Now suppose that we have established the result for systems with K (hierarchical minimax) 
clusters and consider a system with K+l clusters. As above we see that routing errors by JLW 
relating to cluster Ci only act to send additional arrivals to Ci and so the weighted queue 
at each j G Ci eventually has speed at least V|. The system that remains after removing 
Ci has K clusters and initially may lose some arrivals so the maximal drift of any weighted 
queue is bounded above by Vz- As in the two cluster case JLW separates Ci from the rest of 
the system after a finite time and the result follows by induction. ■ 



If |C| = 1 there is nothing to do so we suppose |C| > 2. We consider a bonded sub-cluster 
C C d. This means that there is a n G FI^ such that for any stations j, m G C there is a 
path from j to m in the graph G(7t) with nodes A/d(C) UC and edges {(Si, j) : Si G A/"d(C), j G 



2.7 Proof of Theorem 



C, 7t(i)j > 0}. As C is finite there exists £ > such that 7t(i)j > £ along any such path. 
Similarly there exists A such that Ai(D) > A~ for each Si G Ad(C). 

We modify the quadratic used in Lemma [2] by restricting it to C i.e. we use 




Repeating the calculations from Lemma [2](ii) we have 




isA/b (C) 
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+ 2 y «j( x )(ycWi-i) 

jGC 

where yc = Hjec ^ / w r The even t rates <Xj(x) are bounded uniformly in x so 

2£«jM(YcW)-1) < A 

jec 

for some constant A. Let "f\> = maxj g c w j and note that if Fc(x) > M 2 |C| 2 1& 2 then wiX; — 
w T x T > M for some pair of stations I, r 6 C. 

Suppose that W\X\ — w r x r > M. for some pair of stations I, r e C. As C is bonded there 
is a loop- free path from I to r in the bipartite graph G(7t). Paths in G(tc) have their nodes 
alternately in C and Ad(C) and there must exist a consecutive triple (j, St, )') such that j, 
)' G S t and Wj/Xj/ -WjXj > M/(|C| - 1). Thus 

(xE L (AF c (n) | ^ n = x) < + A 

which is negative for M > A(|C| — 1 )/yc^~£ an d hence the process Fc(£n) is positive recurrent 
by Theorem IH^ii). ■ 
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